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ABSTRACT 


Not all knowledge and skills that educators want to pass to students exists yet in textbooks. 
Some still resides only in the experiences of practicing engineers (e.g., how engineers create new 
products, how designers identify errors in calculations). The critical incident technique, CIT, is an 
established method for cognitive task analysis. It is especially effective for accessing implicit knowl¬ 
edge which exists in a person’s unconscious and is therefore not readily accessible. Such knowledge 
is often referred to as “experience”. The five steps to the critical incident technique are 1) Identify 
your objectives, 2) Make plans and set specifications, 3) Collect the data, 4) Analyze the data, and 
5) Interpret the data and disseminate the results. 

This manuscript presents detailed recommendations on how to conduct a CIT study; therefore, it 
can serve as a “how to” manual for educators who want to obtain experiences from practitioners in 
order to provide those experiences to students as knowledge and skills useful to their professional ca¬ 
reers. This manuscript also includes details of an example application of CIT with lessons learned. 

Keywords: critical incident technique, cognitive task analysis, experience 


MOTIVATION 


Friedman’s book The World is Flat [1] and the National Academy of Engineering’s Gathering 
Above the Rising Storm [2] emphasize the need for changes and imply that some of those changes 
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should be in engineering education. Engineering courses are typically based on skills described in 
textbooks. To meet future needs, however, some of the skills we should be teaching our students 
do not yet exist in a textbook. In some cases, the key skills exist only in the minds of the practitio¬ 
ners developing and using the skills. For example, Dyer et al. [3] identified five skills common to 
extremely successful innovators and claimed that those skills are learned. One of the skills is ob¬ 
serving: scrutinizing common phenomena. But what do innovating engineers look at and look for? 
That information is not yet documented. Another example is the heavy dependence engineers have 
on computers for analysis and design. Experienced engineers have developed skills for evaluating 
those results. Those skills, however, are largely missing from the literature. 

Articulating these skills can be challenging if the practitioner has used them frequently enough 
that they become automatic. In those cases, knowledge of the skills is “implicit knowledge”. The 
information exists in the practitioner’s unconscious and is not easily accessed [4], For the purposes 
of this manuscript we refer to these skills, both explicit and implicit, as “experience” because the 
skills are typically developed on-the-job. 

The process of collecting data about cognitive processes, analyzing it, and communicating the 
information effectively is generally referred to as cognitive task analysis [5], Knowledge elicitation 
techniques for use in cognitive task analysis are well documented in the literature [6], They have 
been used in the fields of weather forecasting, marketing research, nursing, medicine, piloting, 
fire-fighting, mathematics, and military command and control [5-6]. The primary use of knowledge 
elicitation techniques in engineering has been to obtain information from users or customers to 
improve engineering solutions [7-9], However, little exists about using these techniques to gather 
experience from practicing engineers. One rare example is a study by Dawood and Bates [10] who 
used knowledge elicitation techniques to gather information from civil engineers about cost esti¬ 
mating. They summarize the knowledge obtained but provide little detail about how the techniques 
were applied or any lessons learned about knowledge elicitation. 


BACKGROUND 


Cognitive Task Analysis 

Cognitive task analysis, CTA, refers to a variety of research methods with the common objective 
of capturing the following: what someone is thinking about, what that person is paying attention 
to, strategies the person is using to make decisions or identify problems, what the person is trying 
to achieve, and what the person knows about how a process works [5]. According to Crandall et al. 
the primary aspects of CTA are knowledge elicitation, data analysis, and knowledge representation 
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[5], The critical incident technique is just one of the many published CTA methods. For detailed 
descriptions of many CTA methods, refer to Schraagen et al. [6], 

Knowledge elicitation refers to the research methods specifically focused on capturing the 
data. Summaries of documented knowledge elicitation methods list at least 75 different methods 
[5-6,11], The methods use a variety of ways to actually collect the data: interviews, self-reports, 
and observations. Interviews involve a trained interviewer and can be conducted in person or 
via communication technology such as telephone, internet messaging, or video conferencing. 
Self-reports are completed independent of the researcher. This method allows the participant to 
provide the data at a convenient time. However, Wilson notes that people are generally unable 
to provide reliable reports of their own cognitive processes [12], Observations involve record¬ 
ing the actions of the participant while performing the desired tasks. Many of the knowledge 
elicitation methods incorporate interaction between the observer and participant in order to 
record thoughts. 

Throughout the literature it is common to find data analysis and knowledge representation 
discussed with the knowledge elicitation method. Data analysis involves structuring the data and 
discovering meaning. Knowledge representation refers to the process of displaying the data, pre¬ 
senting the findings, and communicating meaning [5], 

Critical Incident Technique 

The critical incident technique, CIT, is a cognitive task analysis method developed by Flanagan 
to identify behaviors that are critical for or detrimental to job performance [13], In this method, the 
interviewer asks the participant to think of a critical incident that determined either the success 
or failure of a task. This incident then becomes the context of the interview. Once the participant 
recalls the incident, the interviewer asks questions designed to obtain the desired data. One of the 
primary advantages of this method is that if the participant considers the incident to be "critical” 
he or she is likely to have a vivid recollection of the incident. The result is that implicit knowledge 
is more likely to be available in conscious memory. The CIT method has been successfully used to 
learn about cognitive processes in a wide range of disciplines: communications, nursing, counseling, 
teaching, medicine, marketing, psychology, and social work [14], 

Flanagan emphasized that the method is flexible, and it has five basic steps: 1) Identify your objec¬ 
tives, 2) Make plans and set specifications, 3) Collect the data, 4) Analyze the data, and 5) Interpret 
the data and disseminate the results [13], These components are consistent with the primary aspects 
of cognitive task analysis discussed in the previous section. 

Flanagan’s goal was to obtain a functional description of an activity [14], Therefore, he started 
by identifying the objectives of the activity. Another approach is to identify the objectives you want 
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to study and let the participant choose the appropriate activity. Either way the objectives must be 
identified first because they impact the other steps. 

The second step, make plans and set specifications, helps ensure consistency between interviews. 
Inconsistencies in results can arise if multiple observers are not all using the same procedures. But 
even a single observer might use inconsistent procedures over time if he or she is not routinely re¬ 
ferring to the plans and specifications. This second step includes determining the types of incidents 
to address in the interviews, developing a protocol for conducting the interviews, and deciding who 
will conduct the interviews. Schluter et ai. emphasized that the interviewer must be properly trained 
in order to ensure that adequate information is obtained [15], 

Data can be collected in step three via interviews, self-reports, or observations. As noted previ¬ 
ously, self-reports might not be effective when the knowledge is implicit. Observations might produce 
the most accurate data. However, if the activity is not routine or the timing of the activity cannot 
be predicted, observations might be ineffective [5], Therefore, the most common way to collect 
data is interviews [5, 15], Interviews allow the interviewer to read non-verbal communication from 
the participant and then probe for more in-depth responses [15]. The participant might require ad¬ 
ditional prompts in order to access implicit knowledge used in the incident. Rous and McCormack 
recommend using the following questions as probes during an interview [16]: 

• What preceded and contributed to the incident? 

• What did the person or people do that had an effect? 

• What was the outcome or result? 

• What made this action effective or ineffective? 

• What could have made this action more effective? 

Flanagan considered the fourth step, data analysis, to be the most important and most difficult [13]. 
Fortunately, there have been many additions to the literature about analyzing verbal data (e.g., [17-19]). 
Meyer and Booker provide recommendations for a “first analysis” of the data in order to identify the 
most important features [20], Schluter et al. recommend inductive analysis which involves two levels 
of interpretation [15]. The first level is iterative review of the individual interviews in order to identify 
themes and subthemes. The second level involves grouping segments of different interviews based 
on similarities or differences. This aids in the development of overarching themes and subthemes. 
Crawford and Signori used inductive analysis with CIT data [21]. They grouped like behaviors into 
“sub-classes” then grouped those into “classes” by similarities. They continued the process by group¬ 
ing similar classes into “sub-areas” and grouping those into a few overarching “major areas”. Someren 
et al. provide detailed instructions on how to code data and perform quantitative analysis on the data 
[22]. Flanagan pointed out, however, that “reporting should be limited to those behaviors which, ac¬ 
cording to competent observers, make a significant contribution to the activity” [13, p. 355]. 


4 


WINTER 2012 





ADVANCES IN ENGINEERING EDUCATION 

The Critical Incident Technique: An Effective Tool For Gathering 
Experience From Practicing Engineers 



The fifth step begins with interpreting how the results might be meaningful in the area of 
study. Flanagan recommended reviewing the first four steps for biases and decisions made 
before interpreting the results [13]. Radford suggests reviewing the relevant literature on the 
topic for suggestions on a framework for interpreting the results [23], The final part of step 5 is 
dissemination. Flanagan made the following recommendations about dissemination: discuss the 
limitations, be explicit about the nature of judgments, and emphasize the value of the results 
[13], Schluter et al. point out that it is important to tailor the presentation of the results to the 
target audience [15], Crandall et al. suggest the following possible products for presentation of 
the results [5]: 

• Textual descriptions 

• Tables, graphs and illustrations 

• Qualitative models such as flowcharts 

• Simulation, numerical, and symbolic models including computer models. 

Additional references on how to present the results in a meaningful way include Miles and 
Huberman [17] and Wolcott [24], 

Because of its versatility and strengths, the critical incident technique has been used for over 
50 years to acquire information about knowledge and cognitive processes. Flanagan emphasized 
that a strength of the method is that it provides a “reporting of facts regarding behavior” rather 
than a “collection of interpretations, ratings, and opinions based on general impressions” [13, p. 355], 
According to Fisher and Oulton, CIT is a “reliable and effective method for gathering rich qualitative 
data for a variety of purposes” [25, p. 125], Studies focused on the reliability and validity of CIT have 
concluded that CIT is indeed a reliable, valid method [26-27], 

Radford pointed out that CIT is “a flexible tool able to be applied in a variety of settings and for 
a variety of purposes” [23, p. 59], Butterfield et al. elaborated on those purposes as ranging from 
“studying effective and ineffective ways of doing something, to looking at helping and hindering 
factors, collecting functional or behavioral descriptions of events or problems, examining successes 
and failures, or determining characteristics that are critical to important aspects of an activity or 
event” [14, p. 476], Flanagan indicated that in the first decade CIT had already been used for the fol¬ 
lowing purposes: developing measures of typical performance, providing benchmarks for measuring 
proficiency, establishing training requirements for a job, developing selection and classification tools 
for employment, tailoring job descriptions to maximize efficiency, studying operating procedures, 
improving the design of equipment for operator interaction, studying motivation and leadership, 
and measuring the effectiveness of counseling and psychotherapy techniques [13], Crandall et al. 
point out that incident-based CTA methods, including CIT, can be used to find the following: cues 
and patterns that experts perceive, rules of thumb experts have devised, the kinds of decisions 
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that experts have to make, the features that make cases typical, and the features of rare cases [5], 
The Society of Consulting Psychology used CIT to identify the significant practitioners, literature, 
and concepts in consulting psychology [28], Fly et al. used CIT to gather faculty experience about 
identifying and dealing with ethical transgressions [29], 

The critical incident technique does have a limitation for gathering practitioner experience though. 
If the desired experience is used routinely, it may be difficult to access the experience through a 
critical incident. Therefore, participants might struggle to provide rich detail [15], A potential remedy, 
however, is to ask about the most recent event that used the experience [5], If the desired experi¬ 
ence occurs over a long period of time (e.g., the entire building design process might take weeks 
or months to complete) the prompts might not be sufficient to access memories rich with details. 
The CIT method might work well, however, to obtain experience from smaller parts of the larger 
experience. A lingering limitation is that critical events for each of the smaller parts might not come 
from a single larger experience. 


DEVELOPING A CRITICAL INCIDENT TECHNIQUE STUDY 


Step 1. Identify Your Objectives 
What are my objectives? 

Begin by reflecting on what you want to be able to do. Then search for published information on 
this topic. Consider not only current textbooks but also out of print texts. Skills or topics considered 
important 50 years ago might no longer be included in current texts, but that information might be 
relevant to your objectives. Another valuable resource is handbooks. These are references typically 
written by practicing engineers for practicing engineers. 

If the information you desire is not published, consider whether it is known by practicing engi¬ 
neers, either known by most practitioners or by a select group. If so, we call it “experience”. Identify 
under what circumstances the experience is utilized. Reflect on whether that experience will tend to 
be stored as explicit or implicit knowledge. Now you probably have sufficient information to select 
an appropriate CTA method. 

Is CIT a good tool for my objectives? 

As noted earlier, the CIT method is useful for studying effective and ineffective ways of perform¬ 
ing a task, identifying helping and hindering factors, gathering descriptions of events or problems, 
examining successes and failures, and determining characteristics that are critical to an activity or 
event [14, p. 476], Some engineering examples include the following: 

• Criteria practitioners use to evaluate competing design options. 
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• The most important factors that practitioners consider in making a decision about which 
material(s) to use. 

• Clues practitioners consider and ignore when troubleshooting problems on a production line. 

• Rules of thumb practitioners use to design in their field. 

• Characteristics of a project that make it typical. 

• Characteristics of a project that require a specialist. 

Step 2. Making Plans and Setting Specifications 
What preliminary work do I need to complete? 

The process starts by developing a plan and submitting it for approval by your Institutional Re¬ 
viewer or Institutional Review Board. We recommend you contact your Institutional Reviewer before 
developing your plan in order to receive pointed guidance on what is expected, what is acceptable, 
and what is not acceptable. 

Whom do I recruit as participants and how? 

If your objective is to gather diverse experiences, plan on interviewing practitioners from a variety 
of companies or firms. Practitioners from the same company are likely to share the same experiences 
and philosophies. When developing the list of companies or firms to approach about the interviews, 
consider the following questions. 

• Might the geographic location of the practitioner have an impact on the types of experience 
or philosophy of design that the practitioner has? If so, try for geographic diversity among 
the companies that you will interview. 

• Might the size of the company have an impact on the types of experience or philosophy of 
design that the practitioner has? If so, try for diversity of company sizes. 

• Might the years of experience or level of education have an impact on the types of experience 
or philosophy of design that the practitioner has? If so, try to interview practitioners with a 
wide range of experience or education level while at one company. Note that we found that 
if the desired data comes best from experiences viewed as “failures”, junior practitioners ap¬ 
pear more comfortable sharing their own “failures”. Senior practitioners, however, tend to have 
observed a greater variety of “failures” in others’ work. 

• Might the gender, race, or ethnicity have an impact on the types of experience or philosophy 
of design that the practitioner has? If so, be sure to include companies with a sufficiently large 
workforce that they will likely have practitioners from under-represented groups. Specifically 
ask the company to include a diversity of participants. 

Before initiating contact with potential participants, create a one page summary of the project. 
Include the type of information you are trying to collect, why the information must be collected 
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from practitioners (e.g., it is not available in the literature), how you will use the results to help 
students, approximately how long each interview will take, and your preference for method of 
interviewing (e.g., questionnaire, over-the-phone, or face-to-face). Also identify the incentive 
for participation (e.g., monetary compensation, copy of results, resulting educational modules). 
Provide this summary with each initial contact. A vague initial request is easily dismissed by a 
busy practitioner. 

Where possible, we initiated contact via phone and followed up via email. In many cases, the 
decision to participate resided with a senior vice president or principal in the firm. Those people 
were best reached initially via email. Consider the following sources for potential participants. 

• Board of Advisors 

• Alumni Office 

• Regional and national professional societies 

• Published lists of top firms 

• Personal contacts 

• Participant recommendations 

How many interviews do I need? 

The key is not the number of interviews but the number of critical incidents reported. The actual 
quantity changes for each study. Butterfield et al. recommend continuing to gather data until the 
additional data consistently fails to result in new categories in the data analysis stage [14], 

Who should conduct the interviews? 

We strongly recommend you chose someone with background in the practitioners’ area of exper¬ 
tise to conduct the interviews. The interviewer does not need to be an expert, but should have basic 
knowledge of the area. Depending on the number of interviews to be conducted, you might choose 
to have multiple interviewers. If so, they must be trained to conduct interviews in a consistent way. 
One option is to have a second interviewer sit in on interviews in the pilot study. Another option is 
to use recordings from the pilot study to train the other interviewers. 

Step 3. Collecting the Data 

How does the interviewer(s) prepare? 

Review the objectives of the study and the script. Brainstorm open-ended questions that might 
be useful in helping participants access and articulate the pertinent information. The most important 
preparation is practice. You can practice with a more experienced interviewer if there is one asso¬ 
ciated with your study. Consider practicing with co-workers or friends who work in the particular 
field of engineering. 
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Create a checklist of items to bring to the interview. Even if the participant is coming to you, the 
checklist reduces the possibility of wasting the practitioner’s time while you get something you 
forgot for the interview. Items to consider packing for interviews include the following: 

• Digital recording device with microphone 

• Extension cord or extra batteries for recording device 

• Extra media storage for recording device 

• Interview script 

• Pad of paper & writing implements 

• Project abstract & release (one for each participant) 

• Business cards 

• Water bottle or coffee mug 

• Breath mints 

If you are visiting the practitioners at their location, you might not know how many participants 
you will be able to interview during the visit. Some engineers might be working on a tight deadline 
and are unable to afford 30 minutes for an interview. We recommend you pack sufficient supplies 
for the most optimistic number of interviews you could conduct during the visit (e.g., all engineers 
working at that location, 15 minutes per person for the full time of your visit). Know ahead of time 
if you can be flexible with your ending time. We found that sometimes key people we wanted to 
interview were not available until after normal business hours. Know ahead of time any restrictions 
you have on whom you want to interview (e.g., only engineers with a specific experience, only en¬ 
gineers with less than a certain number of years experience). 

How do I conduct an interview? 

Dress according to the custom of the location you are visiting. The unofficial dress code for 
practicing engineers can range from jeans to a suit depending on the location. Dressing in the same 
manner as the participants you interview can help make them more comfortable with the process. 
If you are not sure what the typical dress is at a location, ask your host. 

Arrive early. If you are conducting the interview at your location, make sure the room and equip¬ 
ment are set up before the participant is due to arrive. If you are visiting the participant’s location, 
try to arrive just a few minutes early and be prepared to wait patiently for your host. Ask for a loca¬ 
tion where you can set up and have engineers come to you for the interview. The location must be 
private such that no one can overhear the interview. This is to protect the privacy of the participants. 
It also promotes the sharing of information that the participant might find embarrassing or might 
not want their supervisor or co-workers to know they are disclosing. 

Do not assume that your host knows or remembers the background information on your study. 
Therefore, be prepared to explain the information on the project summary sheet in more detail. Do 
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not assume that any of the participants you interview were informed in advance. They might be told 
to report to you without any additional instructions. Therefore, assume that each participant needs 
a brief introduction to the project. Also assume that each participant was told to participate by their 
supervisor. The participant needs to know that participation is voluntary and that you will not share 
what they say with their supervisor, even if they choose to say nothing (i.e., not participate). 

While conducting an interview, we found it best to not take notes. However, do ask as many 
questions as you need to clarify what the participant is describing. Rephrase responses to confirm 
your understanding (e.g., “If I understand you correctly, you ....”). Feel free to ask the participant to 
make sketches to help clarify the information. Add your own notes to the sketch as needed to ensure 
you can understand its relevance to the recorded interview later (e.g., see the clouded notes in the 
two example interview sketches). Be sure to make your notes distinguishable from the participant’s 
notes. This can be done using different color writing implements, boxes or clouds, highlighters, etc. 
Request to keep the original sketch, but offer a copy to the participant if they want one. 

Always finish the interview by asking the participant if they have any questions. Know ahead of 
time what information you can and cannot share. For example, we had permission to share names 
of the participating firms but not individual participants. Expect the participants to be curious as 
to how their experiences will be used. 

Step 4. Analyzing the Data 

What should i do right after the interview? 

Privately, take any additional notes on the discussion that are relevant to the study. If this is not 
possible between interviews, write any notes at the end of the day’s interviews. If you are recording 
only audio during the interviews, it might be worthwhile to note any physical behavior (i.e., body 
language) that suggested the participant was uncomfortable with the interview process. We took 
notes primarily focused on potential data reduction ideas. For example, something the person said 
sounded similar to a previous interview response. Although that would be identified in the later 
analysis, we made notes to help expedite the analysis. We also found that some of the interviews 
generated ideas for topics or techniques for use in the classroom. Rather than hoping those ideas 
would resurface during the data analysis phase, we took notes right away. 

Ideally you can review the recordings of the day’s interviews that evening in order to catalogue 
the data. We stored information about each critical incident in a spreadsheet. The table included 
the time index and file name for each incident so that it could be quickly accessed at any time. We 
stored all demographic and personal information about the participants in another spreadsheet. 
Reviewing the data each day also provides an opportunity for you to critique your interviewing skills 
and make any modifications before the next set of interviews. 


to 


WINTER 2012 





ADVANCES IN ENGINEERING EDUCATION 

The Critical Incident Technique: An Effective Tool For Gathering 
Experience From Practicing Engineers 



What should / wait to do until most or all of the interviews are completed? 

Reviewing the data for categories and themes should wait until you have sufficient incidents 
catalogued. Sufficient is a relative term, but 10 to 20 might be enough to begin identifying com¬ 
monalities. Ideally you will have the latitude to continue gathering data until additional data does 
not add to the categories or themes. Therefore, consider performing data analysis after the addition 
of each group of new data. 

Step 5. Interpreting the Data and Dissemination of Results 
How do / translate the data into something useful? 

After determining the categories or themes, you can review the quantitative data. How often do 
certain categories or themes show up? We recommend focusing on the most frequently cited cat¬ 
egories since they probably represent the experiences most used by practicing engineers. Among 
the most frequently cited categories, evaluate which are best suited for your purpose: classroom 
instruction, professional development seminar, textbook, etc. Some might be outside the scope of 
your purpose. In some cases the higher level cognitive tasks of synthesis and evaluation, as described 
by Bloom and Krathwohl [30], might require experience that is beyond what a student has or can 
develop during the course. Therefore, some categories might need to be learned on the job or as 
part of a continuing education program. 

Once you have identified the categories that are best suited for your purpose, review the original 
data for the entries in these categories. Look for specific skills that can be taught and consider how 
those skills fit into your overall plans. 

What can / share? 

Human subject safeguards will limit some of the details you can share. You will need to consult 
your Institutional Reviewer for specific guidance on your study. However, there are some guidelines 
you can expect. You should be able to share the following: 

• Category or theme headings 

• Aggregated results such as number of incidents in each category 

• Hypothetical examples that would fit into each category 

• Specific skills used 

You might be able to share specific incidents from the interviews if you do not reveal the 
identity of the participant or employer. However, details of the incident must be sufficiently gen¬ 
eral that the actual event is unrecognizable [15]. For example, details on decisions resulting in 
the failure of an o-ring on a manned space flight would be readily recognized as the Challenger 
shuttle incident. 
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What can / publish? 

If you want to publish about your study, you will need to consult your Institutional Reviewer for 
specific guidance. However, there are some guidelines you can expect here as well. In addition to 
the types of information you might want to share in a classroom, you should be able to share the 
following: 

• Objectives of the study 

• List of participating firms or companies, if each has given consent 

• Range of demographics of participants 

• Details of the interview and data analysis procedures 


EXAMPLE APPLICATION OF THE CRITICAL INCIDENT TECHNIQUE 


Description of the Study 

We used the critical incident technique to harvest experience from practicing structural engineers 
about how to identify errors in analysis and design results. Our goal was to identify skills that could 
be taught in undergraduate structural analysis courses. In order to obtain these experiences, we 
conducted interviews with 35 engineers from 10 different firms. 

Preparation for Interviews 

Any research involving human subjects must first be reviewed by the organization’s Institutional 
Reviewer or Institutional Review Board, IRB, depending on the risks involved. For this type of human 
subject research, the biggest concern is protecting the identity of the participants. 

We then began the process of recruiting participants. Our initial contact included a one page 
description of what information we were trying to obtain, why we needed to conduct interviews 
to obtain the information, how the information would be used to help students, and approximately 
how long the interviews would take. We offered to travel to the firm in order to conduct face-to- 
face interviews with as many engineers as possible. We offered no direct compensation for their 
participation, but did offer copies of the aggregated results and any papers that we generated 
from the study. 

We consulted several resources that contained suggestions and checklists for developing and 
administering an interview program [11, 22], One of the recommendations we followed was the use 
of a script (Figure 1). The script ensured that the interviewer followed the protocol approved for 
research involving humans as subjects. It contained the release statement which was also given to 
each participant in written form. Having a script ensured that we asked all of the desired questions 
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Interview Script 

Note: Text in italics is for the interviewer: all other text is to be read aloud. 

Introduction and Permissions: 

Thank you for agreeing to participate in this project. The goal of my research is 
to determine what mental tools experienced structural engineers use to evaluate 
the adequacy of their results or designs, and to develop course materials to teach 
these tools. 

This interview is essential for developing an inventory of mental tools used by 
experienced structural engineers. In order to maintain the integrity of the data, I 
would like to record the interview. Do I have your permission to record this 
interview? [Pause for the answer. Continue only with permission.] 

Neither your name nor the firm’s name will be reported with quotes or anecdotes 
from this interview, so please speak freely. I will point out that quotes, anecdotes 
and summarized data will probably be used in reports and publications, but they 
will not be linked to specific people or firms. 

This interview is one-on-one so that no one in this office will know what we 
discuss. 

I am not here to judge methods used or work you have created. I am not qualified 
to judge those. There is no right answer, so please speak freely. 

Demographic Information: 

[Questions asked of the practitioner] 

1. What is your firm’s name and location? 

2. What is your name? 

3. How many years of experience do you have as a structural engineer? 

4. How many years of experience as a P.E.? 

5. How many years of experience with this firm? 

6. What is the highest degree in structural engineering you have completed? 
Previous Experiences: 

[Use the "Critical Incident Technique ” to link to long term memory of errors and 
checking methods from previous experiences. General aim is to identify 
exemplary behaviors of structural engineers to identify reasonableness of results. 
Key is to ask them to tell me about a specific incident, and the interpretation is 
left to me.] 


Figure 1. Annotated script used to interview practitioners. 
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[Needs to be one-on-one, so participants can speak freely without worrying about 
a supervisor finding out. Interview as many members of the firm as possible since 
it takes only a few minutes.] 

Prompt 1: 

Think of the most recent time you discovered something unreasonable in the 
results of analysis or design. [Pause until participant indicates that has an 
incident in mind.] 

Were you reviewing results of your own or someone else’s? 

Please briefly describe the project. 

What was unreasonable? 

Why was it not reasonable? 

How did you come to notice this result? 

[If the practitioner initially missed something...] 

What contributed to you not catching that initially? [e.g., too busy, new design 
type, inadequate training, shared responsibility] 

Prompt 2: 

Think of the most alarming discovery you have made of something unreasonable. 
[Pause until participant indicates that has an incident in mind.] 

Again, were you reviewing results of your own or someone else’s? 

Please briefly describe the project. 

What was unreasonable? 

Why was it not reasonable? 

How did you come to notice this result? 

[If the practitioner initially missed something...] 

What contributed to you not catching that initially? [e.g., too busy, new design 
type, inadequate training, shared responsibility] 


Figure 1. (Continued) 
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in the desired sequence. For example, the script contained questions for gathering the desired 
background information such as years of experience and level of education. 

The key to accessing a critical incident in the practitioner’s memory is to ask for something that 
is extreme (e.g., most recent, most successful, most important). We refer to this as the memory 
prompt. For this study, we used the following two memory prompts with each participant: 

1. Think of the most recent time you discovered something unreasonable in the results of analysis 
or design. 

2. Think of the most alarming discovery you have made of something unreasonable. 

We chose to not use the term “error” because it has a negative connotation that practitioners 
might be reluctant to admit committing. 

In preparation for the interviews, we practiced asking probing questions that might be used to 
help the participant access the necessary information from memory. Open-ended probing ques¬ 
tions began with phrases such as “ Why did you , “How did you ...”, “What did you..." We avoided 
questions such as “Didyou do this because...” because they might bias the data or elicit only short 
responses of yes or no. We found it useful to have examples of probing questions listed on the script 
as reminders for the interviewer (Figure 1). 

We found it extremely helpful to conduct a pilot study with just a few practitioners relatively 
nearby. This afforded us an opportunity to test the script, memory prompts, and probing questions 
in face-to-face interviews. The script provided in Figure 1 includes the modifications we made after 
the pilot study. 

Conduct of the Interviews 

In order to make the interviews as productive as possible, we chose to have the structural engineer 
member of the research team, Hanson, conduct the interviews with the practitioners. Familiarity with 
the subject proved invaluable during the interviews, because often the practitioners needed probing 
questions in order to access the thought processes used to find the errors. Someone not familiar 
with the subject material would probably have left the interviews with insufficient information. In 
addition, an interviewer who is familiar with the subject will probably share the same vocabulary as 
the practitioner being interviewed. The common language allowed the interviews in this study to 
proceed quickly and stay focused. 

We considered three different approaches to conducting the interviews: face-to-face interviews, 
over-the-phone interviews, and written questionnaires. The pilot study revealed the importance of 
interaction between the interviewer and participant in order to access the desired memories and 
to capture sufficient detail. Therefore, we eliminated the possibility of written questionnaires. The 
pilot study also revealed the importance of sketches to clarify the situation. Although technology 
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exists to share sketches while conducting over-the-phone interviews, we concluded that we would 
obtain the richest data by conducting the interviews face-to-face. 

To capture as much data as possible, we recorded all of the interviews as digital audio files. Video 
recordings require equipment that is more obtrusive, and we felt that video would not add relevant 
data for the study. In addition, we were concerned that the participants would be more self-conscious 
if being video recorded rather than just audio recorded. Each interview required between 15 and 
30 minutes. A recording from an actual interview is available at the following link. 

http://advances.asee.orq/vol03/issue01/media/10-media01.mp3 

We considered taking notes during the interview, but that can be distracting to the practitioner. 
Feedback from the pilot study revealed that the act of jotting something on paper can cause the 
participant to ponder whether he or she said something “wrong”. Therefore, we chose to not take 
notes during the interviews. Instead, the interviewer made any notes in between interviews or at 
the end of the day. The interviewer did, however, collect any sketches generated by the participants 
during the interviews so that the sketches could be referenced when reviewing the audio files. In the 
provided example interview, the participant created two sketches in order to clarify the incidents. 
Those sketches are provided at the following two links. 

http://advances.asee.orq/vol03/issue01/media/10-media02.pdf 

http://advances.asee.orq/vol03/issue01/media/10-media03.pdf 

Analysis of the Verbal Data 

The structural engineer member of the research team, who was the interviewer, reviewed each of 
the interview recordings to distill the pertinent data. The goal of the study was to obtain tools for 
identifying the presence of errors in structural analysis and design results. Therefore, we created a 
table with the following headings: What caught the participant’s attention (the Problem); What hap¬ 
pened to bring about the problem (the Error); How participant knew that there was a problem (the 
Strategy for evaluating); Audio filename; and Time index. While reviewing each audio file, Hanson 
entered the information into the table. 

The CIT interviews resulted in the 35 practitioners identifying 65 instances of errors. Of those 
instances, only once was the practitioner not able to remember or describe how he or she identified 
the error. Once the data from all of the interviews was consolidated into one table, we reviewed the 
strategies for commonalities. We were able to divide the 64 strategies for identifying errors into 
seven categories: comparisons (14), extreme values (3), rules of thumb (7), visualization of load 
path (3), previous experience (19), field (12), and other (6). 

Comparisons are tools where the engineer compares the obtained result with another anticipated 
result (e.g., approximate analysis result, result from another engineer, result from another analysis 
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method or computer program). Extreme values refer to results whose magnitudes are so incredibly 
large or small that the engineer knows they are wrong without comparing with another calculated 
value (e.g., vertical displacement of a beam of 10,000 inches). Rules of thumb are simplified calcula¬ 
tions engineers can make to obtain approximate design parameters (e.g., the depth of a steel beam 
in inches will be roughly the same magnitude as its span length in feet). Visualization of the load 
path refers to engineers creating cross-sections of the structural system and tracing the path that 
load must follow from the point of application until transfer at the foundation. These strategies are 
used to identify locations in the structural system that were not designed to transfer the type of 
internal force present in the system. Previous experience refers to those instances when the engineer 
was able to recognize that a value or condition was not right because the engineer could compare 
the situation to memories (explicit or implicit) of similar, previous projects. The Field category is 
the least desirable because of the cost and potential danger. This category refers to all incidents 
where the error was discovered in the field during or after construction. The final category, Other, 
contains those strategies that did not fit the other six categories. These strategies were primarily 
procedural checks or review procedures. 

Although practicing structural engineers might claim that the ability to find errors is something 
that comes with experience or is innate, analysis of the verbal data showed something very different. 
The first four categories are strategies that can be taught at the undergraduate level: comparisons, 
extreme values, rules of thumb, and visualization of load path. They accounted for 42% of the in¬ 
stances identified by the practicing structural engineers. Of the other categories, only the previous 
experience category (30% of the instances) must wait to be developed on the job over time. 

How the Information Impacted Classroom Instruction 

This study focused specifically on using those experiences that were relevant to the Structural 
Analysis I and II courses. The Structural Analysis I course is mandatory for all civil engineering majors 
and is typically taken in the junior year. The Structural Analysis II course is an elective recommended 
for those civil engineering majors with an interest in structural design as a career. Both courses are 
taught by the structural engineer member of the research team, Hanson. Because we narrowed 
the focus to identifying errors in structural analysis results, Hanson focused on strategies in the 
comparisons and extreme values categories. Rules of thumb are transferable to the classroom, and 
they are appropriate in design courses. Although visualizing the load path is related to structural 
analysis, the sources of error found with these strategies originate in design decisions. Therefore, 
Hanson chose to not emphasize those strategies in the analysis courses. The checking and review 
procedures contained in the Other category are typically taught as part of orientation to the firm 
or design office; therefore, Hanson chose to not spend time on these strategies either. 
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Strategies within the extreme values category were easiest to implement in the classroom. With 
only a few example problems, the students were able to develop sufficient experience to recognize 
a range of realistic orders of magnitude for answers. The key was to make the students explicitly 
aware of these experiences and the ranges. As the instructor, Hanson periodically and intentionally 
committed an error in calculations during class that resulted in an answer several orders of magni¬ 
tude outside the expected range. If none of the students challenged the answer, Hanson challenged 
them to argue whether it was within the reasonable range or not. For further reinforcement, Hanson 
made supplemental deductions on homework, project assignments, and exams if students submit¬ 
ted errors in the extreme value range. 

The majority of the strategies that Hanson incorporated into the courses came from the com¬ 
parisons category. In order to identify relevant strategies from this category, Hanson returned to 
the data from the interviews. Three subcategories pertinent to classroom instruction emerged from 
this review. 

1. Fundamental principles 

2. Approximations 

3. Features of the solution 

Fundamental principles refer to the scientific principles that underlie engineering. In the case of struc¬ 
tural analysis, the students learn the fundamental principles in the Statics and Mechanics of Materials 
courses, which are prerequisites for the Structural Analysis I course. The point emphasized to the 
students is that no matter how complex the model and no matter what assumptions they make, the 
fundamental principles must still be satisfied. The approximation skills taught in these courses fall into 
two subcategories: simplified loading and assumptions. Sometimes the situation can be analyzed by 
hand calculations using the fundamental principles, but the loading conditions are so complex that 
the calculations are prone to errors. In those cases, replacing the complex loading with a simplified 
version allows for fast calculation of an approximate solution that can be compared to the solution 
for the complex loading. One of the primary reasons that practitioners use computer programs to 
perform structural analysis is that most structures are indeterminate. This means that there are more 
unknown internal forces than equations of equilibrium. To convert the situation back to one where 
hand calculations can be used, practitioners sometimes make assumptions about the internal forces. 
Common examples are the “portal method” and “cantilever method” for analyzing rigid frames sub¬ 
jected to lateral loads. By considering boundary conditions, continuity, and fundamental principles, 
engineers can anticipate features of a solution. These skills are especially helpful when reviewing 
graphical results such as internal force diagrams, influence lines, and deflected shapes. 

As a measure of the impact of teaching these strategies to undergraduates, the authors used 
a specially developed set of final exam questions. Each question described a structural analysis 
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Term 

Number 

Correctly Identify 
Most Reasonable 

Correctly Explain Why 
Reasonable 

Fall 2004 

34 

50% 

49% 

Fall 2005 

48 

67% 

70% 

Fall 2006 

44 

66% 

69% 

Fall 2007 

30 

69% 

69% 

Fall 2008 

42 

69% 

73% 

Fall 2009 

44 

65% 

62% 

Fall 2010 

31 

65% 

67% 

Practitioners 

8 

92% 

72% 


Table 7; Performance of students and practitioners on the Structural Analysis i final exam. 


Term 

Number 

Correctly Identify 
Most Reasonable 

Correctly Explain Why 
Reasonable 

Winter 2004 

9 

35% 

46% 

Winter 2005 

7 

43% 

44% 

Winter 2006 

17 

49% 

65% 

Winter 2007 

11 

48% 

71% 

Winter 2008 

16 

59% 

68% 

Winter 2009 

16 

57% 

68% 

Practitioners 

8 

65% 

58% 


Table 2: Performance of students and practitioners on the Structural Analysis II final 
exam. 


problem and provided four answers. The students had to identify the most reasonable answer, and 
they had to justify their choice. The resulting performance for the Structural Analysis I course is 
shown in Table 1. Table 2 presents the results from the Structural Analysis II course. The first class 
in each table was the control group. They were not taught about the strategies used by practicing 
engineers. As a comparison, the same test questions were given to several of the practicing engi¬ 
neers. Their performance is recorded in the last line in each table. 
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The results demonstrate that the information gathered from practicing structural engineers can 
be used effectively in an undergraduate course. In the case of the Structural Analysis I course, the 
students who learned about the strategies used by practitioners were able to identify the most 
reasonable answer 15-19% more frequently than the control group. In addition, those students 
had elevated their ability to explain their decisions to roughly the same level of practitioners. The 
impact in the Structural Analysis II course is even more pronounced. There the students were able 
to identify the most reasonable answer almost as frequently as the practitioners, and the students 
were consistently able to explain their decisions better than the practitioners. Specific details of 
the approach used in the classroom, homework format, and additional data are provided in other 
references by the authors [31-32], 


LESSONS LEARNED 


Through reviews of the literature and experience with this study, we developed the following list 
of potential challenges and recommendations. 

• The descriptions of the incidents by the practicing engineers typically required clarification to 
ensure full understanding of the details of what happened and why. In addition, the practicing 
engineers used terminology common to their profession. Therefore, choosing an interviewer 
familiar with the subject matter was essential to the success of each interview. 

• Some people may be hesitant to share certain information even if it is explicitly available in 
their memory. In this study, one potential participant was unwilling to acknowledge any er¬ 
rors, even ones identified and fixed as part of the routine design process. That engineer’s only 
response was “We are very careful. We don’t make mistakes.” Trying to force a participant to 
provide information that he or she does not want to share might yield inaccurate or incomplete 
data. Therefore, rather than trying to overcome the resistance of that engineer, the interviewer 
terminated the interview and moved to the next engineer. 

• We did not use the term “critical incident” in the interviews. Schluter et al. found that participants 
each have different definitions of “critical” and “incident” and, therefore, often claimed that they 
had not experienced an “incident” even though they had experienced a relevant situation [15], 

• Before embarking on a large scale interview program, we conducted a pilot study with a few 
practicing engineers. This allowed the interviewer to clarify any ambiguities in the script, 
confirm the usefulness of the critical incident prompts, and practice asking probing questions 
that do not lead the practitioner toward a preferred answer. Use of a pilot study is consistent 
with the recommendations in the literature [33-34], 
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• The interviews provide a great opportunity to obtain additional information that might be 
useful to your undergraduate program and/or your students. After the CIT interview, the 
practicing engineers were typically willing to spend a few minutes sharing their thoughts on 
undergraduate education. Some questions that might be asked include the following: 

- What topic in your undergraduate education was most helpful? 

- What single topic would have been most helpful if it had been added to your undergraduate 
education? 

- If you could provide a single piece of advice to undergraduates preparing for a career in 
your field, what would it be? 

• It is helpful to review and catalogue the data as soon as possible after the interview rather 
than waiting for all interviews to be complete. That evening or the next day is ideal. This serves 
four purposes. 1) It makes the cataloguing process less onerous. 2) It is an opportunity for the 
interviewer to record thoughts about the interview while the memories are still fresh. 3) The 
interviewer can confirm adherence to the protocol for consistency. 4) The interviewer can 
perform self-assessment about the probing questions used in order to delete, add or modify 
questions before the next set of interviews. 


SUMMARY 


The critical incident technique is one example of a Cognitive Task Analysis method for obtaining 
experience from “experts”. CIT has been successfully used for over 50 years in a variety of fields; 
however, little yet exists about its use in engineering. Therefore, we created this manuscript as a 
“how-to” manual for obtaining experience from practicing engineers. As an example of using CIT 
with engineers, the authors were able to gather experience from practicing structural engineers, 
identify teachable strategies, and impact student performance. By combining lessons learned from 
the example study with guidance from the literature, we have created a unique resource for engi¬ 
neering educators and researchers. 
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